Methods for acquiring or calculating a depth image or motion image which represents the distribution of depth information or motion information from a photographed image have been proposed. The depth information or motion information acquired from the photographed image, however, often includes an error. For example, in the case of determining the depth information or motion information by acquiring the correspondence of two images using a template matching technique, a major error could be generated in a boundary of objects in the images. Such an error is generated when one template includes objects of which depths and motions are different. The acquired depth information or motion information often becomes incorrect information, the depth or motion value of which is an intermediate value between the distances or motion values of these objects. The size of the region having the error depends on the size of the template. A similar error could be generated in the boundary of the object, even in the case of determining the depth information or motion information using a method other than the template matching technique.
Examples of a method for correcting information of each pixel of the depth image or motion image including an error are as follows.
In Patent Literature 1, a confidence coefficient of depth information of each pixel is calculated, and the depth information is corrected using this confidence coefficient. The confidence coefficient of the depth information is calculated based on the magnitude of the brightness value in the photographed image, the magnitude of the change of the brightness, frequency characteristic or motion information. This allows to make the confidence coefficient to be small in an area where there is no texture or in an area where motion is large. However, if the object boundary of each object has texture and motion thereof is small, the confidence coefficient is calculated high in the object boundary. In other words, according to the method of Patent Literature 1, the confidence coefficient is calculated without considering the object boundary.
In Patent Literature 2, the depth information is corrected as follows. First, clustering is performed based on the pixel values of the photographed image or the depth values of the depth image, and each pixel is classified into a plurality of classes. The class of the correction target pixel is determined using a pixel value statistic (e.g. mean value), the correction target pixel, and the pixel value of the pixel periphery to the correction target pixel. Then the depth value of the correction target pixel is replaced with a representative depth value (e.g. mean value of depth values within the class). By this processing, correction can be performed considering the spatial continuity of the pixel values. However if an error is included in the depth values in the class, the representative depth value in the class is shifted from the correct depth value, and correction becomes insufficient.
In Non-patent Literature 1, a depth image is corrected by a weighted cross-bilateral filter using depth information, brightness information of a photographed image, and confidence coefficient information derived from these pieces of information. In other words, a peripheral pixel, of which difference of the depth value or brightness is large compared with the correction target pixel, is regarded as unreliable, and is not used for the correction processing. However, the confidence coefficient calculated like this is not the confidence coefficient of the depth information itself, but is simply a relative confidence coefficient among the pixels. Further, according to this method, the correction target pixel that includes an error has a large depth difference compared with the peripheral pixels that does not include an error, and therefore correct depth information of the peripheral pixels that does not include an error cannot be used for correction processing. Patent Literature 3 discloses that the confidence coefficient of the depth value, when the depth image is encoded, is determined.